Method and apparatus for displaying an augmented-reality image corresponding to a microscope view

ABSTRACT

A method of displaying an augmented reality (AR) image includes controlling an AR display to display the AR image as being at least partially overlaid over a view through a microscope while the view is visible to a user of the microscope, wherein the AR image includes a cursor; receiving a user input from a user interface; setting an anchor point based on the user input; updating the AR image to include the anchor point; controlling the AR display to display the updated AR image; detecting a motion of a slide of the microscope; adjusting the AR image by moving at least one from among the cursor and the anchor point within the AR image based on the detected motion; and controlling the AR display to display the adjusted AR image.

FIELD

The present disclosure is related to augmented reality. Specifically,the present disclosure is related to displaying an augmented-realityimage corresponding to a microscope view.

BACKGROUND

Currently, most microscope operators purely rely on their experience togive rough measurement, such as the size of the lesion area or thenumber of cells. This approach is objective and thus inaccurate.

Some solutions propose taking digital photos from the microscope firstand then use computer software to conduct analysis. For instance,CN109029247A and EP2473928B1 propose using software to measure photostaken by special light microscope.

However, to conduct measurement on computer, users need to switchbetween microscope and computer to operate, which is not efficient andtime consuming. As a result, such systems have been rarely adopted byusers such as pathologists.

Also, because the measurement is conducted on the screenshot ofmicroscope, the area that can be measured is limited by the field ofview under microscope. If the tissue is too large to be fully capturedwith one screenshot, it cannot be measured by the system.

SUMMARY

According to an embodiment, a method of displaying an augmented reality(AR) image, the method may include controlling an AR display to displaythe AR image as being at least partially overlaid over a view through amicroscope while the view is visible to a user of the microscope,wherein the AR image includes a cursor; receiving a user input from auser interface; setting an anchor point based on the user input;updating the AR image to include the anchor point; controlling the ARdisplay to display the updated AR image; detecting a motion of a slideof the microscope; adjusting the AR image by moving at least one fromamong the cursor and the anchor point within the AR image based on thedetected motion; and controlling the AR display to display the adjustedAR image.

According to an embodiment, a device for displaying an augmented reality(AR) image may include an AR display configured to display an AR imageas being at least partially overlaid over a view through a microscopewhile the view is visible to a user of the microscope; at least onemotion sensor configured to detect a motion of a slide of themicroscope; a user interface configured to receive a user input; atleast one memory configured to store program code; and at least oneprocessor configured to read the program code and operate as instructedby the program code, the program code including: first displaying codeconfigured to cause the at least one processor to control the AR displayto display the AR image including a cursor; receiving code configured tocause the at least one processor to receive the user input from the userinterface; setting code configured to cause the at least one processorto set an anchor point based on the user input; updating code configuredto cause the at least one processor to update the AR image to includethe anchor point; second displaying code configured to cause the atleast one processor to control the AR display to display the updated ARimage; detection code configured to cause the at least one processor todetect the motion of the slide from the motion sensor; adjusting codeconfigured to cause the at least one processor to adjust the AR image bymoving at least one from among the cursor and the anchor point withinthe AR image based on the detected motion; and third displaying codeconfigured to cause the at least one processor to control the AR displayto display the adjusted AR image.

According to an embodiment, a non-transitory computer-readable mediummay store instructions, the instructions including: one or moreinstructions that, when executed by one or more processors of a devicefor displaying an augmented reality (AR) image, cause the one or moreprocessors to: control an AR display to display the AR image as being atleast partially overlaid over a view through a microscope while the viewis visible to a user of the microscope, wherein the AR image includes acursor; receive a user input from a user interface; set an anchor pointbased on the user input; update the AR image to include the anchorpoint; control the AR display to display the updated AR image; detect amotion of a slide of the microscope; adjust the AR image by moving atleast one from among the cursor and the anchor point within the AR imagebased on the detected motion; and control the AR display to display theadjusted AR image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of an overview of an example implementation of anaugmented-reality microscope system, according to an embodiment;

FIGS. 2A-2B are illustrations of a movement of a slide of anaugmented-reality microscope system, according to an embodiment;

FIG. 3 is a flow chart of an example process for displaying anaugmented-reality image.

FIG. 4 is a diagram of an example environment in which systems and/ormethods, described herein, may be implemented;

FIG. 5 is a diagram of example components of one or more devices ofFIGS. 4; and

DETAILED DESCRIPTION

Augmented reality (AR) may refer to a technology that superimposes acomputer-generated image on a user's view of the real world, thusproviding a composite view. Embodiments of the present disclosure relateto processes, devices, and systems that may allow users to directlymeasure objects under a microscope using an AR image overlaid onto aview through the microscope. According to embodiments, a user can plotvirtual objects such as a point, line, circle, rectangle, or a regionwith arbitrary shape directly under microscope. Embodiments may allow auser to generate measurement based on a user's plot, such as the lengthof a line, the size of a region, the intensity histogram in the region,the type of the region, or the number of objects, for example cells,inside the regions. The virtual objects may also be saved and applied tohighlight areas on microscope screenshot.

Embodiments of the present disclosure allow a user to conductmeasurement and show the result directly under a microscope. Users mayfreely move the tissue around during measurement and thus limitations onthe field of view may be reduced. Users may also use this system to plotand highlight a region of interest under microscope.

FIG. 1 is an illustration of an embodiment of an AR microscope system100. According to embodiments, AR microscope system 100 may include animage sensor, for example a camera 102, which may capture a microscopeimage 104 of a view under microscope 106. A computer 108 may compute andgenerate the graphics, for example an AR image 110, based on themicroscope image 104 captured by the camera 102. An AR device 112 mayoverlay the computer generated AR image 110 with the view under themicroscope 104. A user may observe the slide 114 using microscopetogether with the AR image 110 as AR overlaid view 116. In this way, alarge amount of information may be presented through microscope 106. Inaddition, the user may interact with computer 108 by speech controlthrough microphone 118, or physical input devices including but notlimited to button 120, or a keyboard, a mouse, or a paddle.

When conducting measurement, a virtual cursor 122 will be shown as a dotin the center of AR device 108. The user can place anchor points 124using this virtual cursor 122. The virtual cursor 116 may be similar to,for example, a cursor shown on computer monitor. However, in embodiment,virtual cursor 116 may stay in the middle of the view through microscope104 while the slide 110 moves. This may be consistent with the operationof microscope 106. In embodiments, slide 110 may be placed on top of thestage 126 and the operator move the stage 126 around to inspectdifferent parts of the slide 110. When moving slide 110, all previousplots such as anchor points 124 may move together with the slide.

In embodiments, AR microscope system 100 may include microscope 104,which may zoom and inspect slide 114. The microscope 104 can send asignal to computer 108 to tell the computer 108 that objective lens ofmicroscope 104 is currently in use when the objective lens is switchedon, or the computer 108 send a request signal for the information. ARmicroscope system 100 may include an image sensor such as digital camera102, which may be installed on a lens tube of microscope 104 to capturean image 106 of the view under microscope 104. AR microscope system 100may include an AR device 112 such as an AR lens, which may be installedon an ocular portion of microscope 104 to overlay a virtual screen suchas AR image 110 on top of the view of slide. AR microscope system 100may include computer 108, which may provide image computation and datastorage.

According to embodiments, AR microscope system 100 may includeMicrophone 118, which may be a voice input device for user to control ARmicroscope system 100. AR microscope system 100 may also include otherinputs such as button 120, or a keyboard, mouse, or paddle, which may bephysical input devices for user to control the AR microscope system 100.

According to embodiments, AR microscope system 100 may include softwareand graphical user interfaces. For example, AR microscope system 100 amotion tracking algorithm track the movement of slide 114 undermicroscope 104. AR microscope system 100 may include a virtual layer,for example a layer shown in AR image 110, that may move together withthe slide 110 and record virtual objects such as anchor point 124plotted by the user. AR microscope system 100 may include virtual cursor122 placed in the center of the view. AR microscope system 100 mayinclude a user interface that enable users to plot under microscope 104,for example by drawing line 128. AR microscope system 100 may include auser interface that can interactively show the measurement result 130under microscope 104. AR microscope system 100 may include a storagesystem that can save images and videos captured by the camera 102. Astorage system that can save the virtual plots, for example as shown inAR image 110, which are made by the user. AR microscope system 100 mayinclude a computer software that allows users to load the saved imagesand videos together with the virtual plots. AR microscope system 100 mayinclude a printing service that can print the captured photos with orwithout virtual plot overlaid on top of it.

As illustrated in FIGS. 2A-2B, a user can experience the operation of ARmicroscope system 100 as if there is a virtual screen, for example ARimage 110, that moves together with the slide 114. And all the plots,for example anchor point 124 and line 128 may be drawn on the virtualscreen. For example, FIG. 2A shows an example of slide 114, AR image110, and AR overlaid view 116 before a motion of slide 114 in theindicated moving direction, while FIG. 2B shows an example of slide 114,AR image 110, and AR overlaid view 116 are the motion of slide 114 inthe indicated moving direction.

According to an embodiment, computer 108 may track a motion of slide114. For example, camera 102 may record microscope view 106 in asequence of frames. The frames are passed to computer 108. The computer108 may compare adjacent frames and detect and compute the shift fromthe previous frame F_(t-1) to current frame F_(t).

One possible way to compute such a motion or shift is in followingsteps:

1) Compute key points P_(t-1) on frame F_(t-1)2) Find the correspondence of P_(t-1) on the current frame: P_(t)3) Compute the distance between P_(t-1) and P_(t):

Δ_(t) =P _(t) −P _(t-1)

4) Shift the virtual screen with Δ_(t) and update the graphical plotincluded in AR image 110 of AR display 112.

In embodiments, the motion tracking algorithm can also been implementedin other ways. For example, the motion tracking algorithm may include anoptical flow algorithm or end-to-end deep learning Siamese network ormultiple model ensemble for to estimate the motion between two adjacentframes to improve the accuracy. In addition, the motion trackingalgorithm may consider the movement across multiple previous frames toimprove temporal stability.

According to embodiments, a user may draw a virtual plot such as line128 on AR image 110. For example, line 128 may include a ruler, anarrow, or any other visual representation as desired. In an embodiment,a user may use speech command (e.g. say “draw a line”) or keyboardcommand (e.g. press key ‘l’) or press the button 120 with predefinedpattern (e.g. single click) to start. When started, AR microscope system100 may record the current location of virtual cursor 122 by placinganchor points 124 on the virtual screen 110 accordingly. When moving thestage 114, a line 128 may be plotted between the virtual cursor 122 andthe anchor point 124. If the user chooses a ruler function, ameasurement such as a length of the line may be shown next to thevirtual cursor 122. If the user chooses an arrow function, an arrowpointing from virtual cursor 122 to the anchor 124 may be shown. Theuser can finish and save the plot by speech command (e.g. say “save”) orkeyboard command (e.g. press key ‘s’) or by pressing the button 120 witha predefined pattern (e.g. double click). The user can finish and deletethe plot by speech command (e.g. say “stop”) or keyboard command (e.g.press key ‘x’) or press the button 120 with predefined pattern (e.g.long press).

To conduct measurements, AR microscope system 100 may consider thephysical size of pixels of the camera 102 and the object lens. This canbe manually set, or can be automatically corrected by a standard gridruler slide together with an auto-correction program. The grid ruler maybe a slide with checker-board like grid plot on it. Each square on thecheck board may be of fixed size: e.g. 10 micron meter. Theauto-correction program may detect those grids and count the number ofpixels inside the grids to automatically estimate the physical size ofpixels.

According to embodiments, a user may draw a virtual polygon plot on ARimage 110. In an embodiment, the user may use speech command (e.g. say“draw a polygon”) or keyboard command (e.g. press key ‘p’) or press thebutton with predefined pattern (e.g. double click) to start. Whenstarted, AR microscope system 100 may record the current location ofvirtual cursor 122 by placing an anchor points 124 on the virtual screen110 accordingly. When moving the stage 114, a line 128 may be plottedbetween the virtual cursor 122 and the anchor point 124. The user candrop a new anchor point in current virtual cursor location by speechcommand (e.g. say “next”) or keyboard command (e.g. press key ‘k’) orpress the button 120 with predefined pattern (e.g. single click). Whenthe user does so, a fixed line may be plotted between the new anchorpoint and the previous anchor point, and another line may start from thenew anchor to the virtual cursor 122. The user may finish and save theplot by speech command (e.g. say “save”) or keyboard command (e.g. presskey ‘s’) or press the button 120 with predefined pattern (e.g. doubleclick). When the user does so, a line may be plotted between the firstcursor and the last cursor to generate a closed polygon. After the userfinishes, the area size of the polygon can be shown on the virtualscreen 110 next to the polygon. The user can finish and delete the plotby speech command (e.g. say “stop”) or keyboard command (e.g. press key‘x’) or press the button 120 with predefined pattern (e.g. long press).

According to embodiments, a user may draw a spherical or rectangularplot on AR image 110. n an embodiment, the user may use speech command(e.g. say “draw a sphere”) or keyboard command (e.g. press key ‘o’) orpress the button 120 with predefined pattern (e.g. long press) to start.When started, AR microscope system 100 may record the current locationof virtual cursor 122 by placing anchor points 124 on the virtual screen110 accordingly. When moving the stage 114, a sphere may be plotted bytaking the anchor point 124 as center and the line 128 between thevirtual cursor 122 and the anchor point 124 as radius. If the userchooses a rectangle function, a rectangle may be plotted by taking theline 128 between virtual cursor 122 and the anchor point 124 asdiagonal. The user can finish and save the plot by speech command (e.g.say “save”) or keyboard command (e.g. press key ‘s’) or press the button120 with predefined pattern (e.g. double click). The user can finish anddelete the plot by speech command (e.g. say “stop”) or keyboard command(e.g. press key ‘x’) or press the button with predefined pattern (e.g.long press).

According to embodiments, a user may take measurements. Measurements,for example measurement 130, can be shown during the plot or after theplot finished. The user may turn measurements on or off, or select themeasurement to be displayed by the speech or physical controls. The typeof measurements available includes but are not limited to:

1) Length: line length, circle radius, rectangle diagonal2) Size: area size of an object3) Color distribution: color mean/std, color histogram, frequencydistribution4) Cell count: the number of cells inside an area5) Tissue count: the percentage of different type of tissues inside anarea

Accordingly, AR microscope system 100 may allow the user conductinteractive plots under microscope to high-light and/or measure areasdirectly under microscope, which the traditional microscope is notcapable of. AR microscope system 100 may allow the user to measureobjects under microscope even the when the object is too large to beentirely captured under microscope, which current microscope software isnot capable of. Further, AR microscope system 100 may allow the user tosave and/or print AR images such as plots together with the imagescaptured, which helps record the data for later inspection.

FIG. 3 is a flow chart of an example process 300 for displaying anaugmented reality (AR) image. In some implementations, one or moreprocess blocks of FIG. 3 may be performed by computer 108 or platform420, described below. In some implementations, one or more processblocks of FIG. 3 may be performed by another device or a group ofdevices separate from or including computer 108 or platform 420, such asmicroscope 104, camera 102, AR device 112, or user device 410 describedbelow.

As shown in FIG. 3, process 300 may include controlling an AR display todisplay the AR image as being at least partially overlaid over a viewthrough a microscope while the view is visible to a user of themicroscope, wherein the AR image includes a cursor (block 310).

As further shown in FIG. 3, process 300 may include receiving a userinput from a user interface (block 320).

As further shown in FIG. 3, process 300 may include setting an anchorpoint based on the user input (block 330).

As further shown in FIG. 3, process 300 may include updating the ARimage to include the anchor point (block 340).

As further shown in FIG. 3, process 300 may include controlling the ARdisplay to display the updated AR image (block 350).

As further shown in FIG. 3, process 300 may include detecting a motionof a slide of the microscope (block 360).

As further shown in FIG. 3, if there is no motion (block 360—NO), thenprocess 300 may return to block 310.

As further shown in FIG. 3, if motion is detected (block 360—YES), thenprocess 300 may include adjusting the AR image by moving at least onefrom among the cursor and the anchor point within the AR image based onthe detected motion (block 370).

As further shown in FIG. 3, process 300 may include controlling the ARdisplay to display the adjusted AR image (block 380).

According to an embodiment the adjusted AR image may include a linedisplayed between the anchor point and the cursor.

According to an embodiment, process 300 may further include defining anarea within the AR image based on the line.

According to an embodiment the anchor point may be included in aplurality of anchor points, the line may be included in a plurality oflines determined based on the plurality of anchor points, and the areamay be bounded by the plurality of lines.

According to an embodiment the area may include a rectangle, and theline may be a diagonal of the rectangle.

According to an embodiment the area may include a circle, the cursor maybe located at a center of the circle, the anchor point may be located ona circumference of the circle, and the line may be a radius of thecircle.

According to an embodiment the adjusted AR image may include ameasurement determined based on the line, wherein the measurement mayinclude at least one from among a length of the line, a size of thearea, a color distribution of colors included in the area, a colorhistogram of the colors included in the area, a frequency distributionof the colors included in the area, a number of cells included in thearea, and a percentage of the area that may include a predetermined typeof tissue.

According to an embodiment the user input may include at least one fromamong a button press, a keyboard input, a mouse input, and a voiceinput.

According to an embodiment process 300 may further include capturing aplurality of image frames of the view through the microscope, wherein adistance of the motion and a direction of the motion may be determinedbased on a difference between a first point in a first image frame of aplurality of image frames and a second point in a second image frame ofthe plurality of image frames, the second point corresponding to thefirst point, and wherein the adjusted AR image may be adjusted based onthe determined distance and the determined direction.

According to an embodiment process 300 may further include storing animage comprising the view through the microscope at least partiallyoverlaid with the adjusted AR image.

Although implementations herein describe phoneme sequences, it should beunderstood that other implementations include word sequences, charactersequences, and/or the like, as intermediate sequences. In other words,other implementations include the direct mapping between a speechwaveform and word and/or character sequences.

Although FIG. 3 shows example blocks of process 300, in someimplementations, process 300 may include additional blocks, fewerblocks, different blocks, or differently arranged blocks than thosedepicted in FIG. 3. Additionally, or alternatively, two or more of theblocks of process 300 may be performed in parallel.

FIG. 4 is a diagram of an example environment 400 in which systemsand/or methods, described herein, may be implemented. As shown in FIG.4, environment 400 may include a user device 410, a platform 420, and anetwork 430. For example, user device 410 may correspond to variouscomponents such as microscope 104, camera 102, and AR device 112. Inaddition, platform 420 may correspond to computer 108. Devices ofenvironment 400 may interconnect via wired connections, wirelessconnections, or a combination of wired and wireless connections.

User device 410 includes one or more devices capable of receiving,generating, storing, processing, and/or providing information associatedwith platform 420. For example, user device 410 may include a computingdevice (e.g., a desktop computer, a laptop computer, a tablet computer,a handheld computer, a smart speaker, a server, etc.), a mobile phone(e.g., a smart phone, a radiotelephone, etc.), a wearable device (e.g.,a pair of smart glasses or a smart watch), or a similar device. In someimplementations, user device 410 may receive information from and/ortransmit information to platform 420.

Platform 420 includes one or more devices capable of generating an ARimage to be displayed as overlaid onto a view through a microscope, asdescribed elsewhere herein. In some implementations, platform 420 mayinclude a cloud server or a group of cloud servers. In someimplementations, platform 420 may be designed to be modular such thatcertain software components may be swapped in or out depending on aparticular need. As such, platform 420 may be easily and/or quicklyreconfigured for different uses.

In some implementations, as shown, platform 420 may be hosted in cloudcomputing environment 422. Notably, while implementations describedherein describe platform 420 as being hosted in cloud computingenvironment 422, in some implementations, platform 420 is not becloud-based (i.e., may be implemented outside of a cloud computingenvironment) or may be partially cloud-based.

Cloud computing environment 422 includes an environment that hostsplatform 420. Cloud computing environment 422 may provide computation,software, data access, storage, etc. services that do not requireend-user (e.g., user device 410) knowledge of a physical location andconfiguration of system(s) and/or device(s) that hosts platform 420. Asshown, cloud computing environment 422 may include a group of computingresources 424 (referred to collectively as “computing resources 424” andindividually as “computing resource 424”).

Computing resource 424 includes one or more personal computers,workstation computers, server devices, or other types of computationand/or communication devices. In some implementations, computingresource 424 may host platform 420. The cloud resources may includecompute instances executing in computing resource 424, storage devicesprovided in computing resource 424, data transfer devices provided bycomputing resource 424, etc. In some implementations, computing resource424 may communicate with other computing resources 424 via wiredconnections, wireless connections, or a combination of wired andwireless connections.

As further shown in FIG. 4, computing resource 424 includes a group ofcloud resources, such as one or more applications (“APPs”) 424-1, one ormore virtual machines (“VMs”) 424-2, virtualized storage (“VSs”) 424-3,one or more hypervisors (“HYPs”) 424-4, or the like.

Application 424-1 includes one or more software applications that may beprovided to or accessed by user device 410. Application 424-1 mayeliminate a need to install and execute the software applications onuser device 410. For example, application 424-1 may include softwareassociated with platform 420 and/or any other software capable of beingprovided via cloud computing environment 422. In some implementations,one application 424-1 may send/receive information to/from one or moreother applications 424-1, via virtual machine 424-2.

Virtual machine 424-2 includes a software implementation of a machine(e.g., a computer) that executes programs like a physical machine.Virtual machine 424-2 may be either a system virtual machine or aprocess virtual machine, depending upon use and degree of correspondenceto any real machine by virtual machine 424-2. A system virtual machinemay provide a complete system platform that supports execution of acomplete operating system (“OS”). A process virtual machine may executea single program, and may support a single process. In someimplementations, virtual machine 424-2 may execute on behalf of a user(e.g., user device 410), and may manage infrastructure of cloudcomputing environment 422, such as data management, synchronization, orlong-duration data transfers.

Virtualized storage 424-3 includes one or more storage systems and/orone or more devices that use virtualization techniques within thestorage systems or devices of computing resource 424. In someimplementations, within the context of a storage system, types ofvirtualizations may include block virtualization and filevirtualization. Block virtualization may refer to abstraction (orseparation) of logical storage from physical storage so that the storagesystem may be accessed without regard to physical storage orheterogeneous structure. The separation may permit administrators of thestorage system flexibility in how the administrators manage storage forend users. File virtualization may eliminate dependencies between dataaccessed at a file level and a location where files are physicallystored. This may enable optimization of storage use, serverconsolidation, and/or performance of non-disruptive file migrations.

Hypervisor 424-4 may provide hardware virtualization techniques thatallow multiple operating systems (e.g., “guest operating systems”) toexecute concurrently on a host computer, such as computing resource 424.Hypervisor 424-4 may present a virtual operating platform to the guestoperating systems, and may manage the execution of the guest operatingsystems. Multiple instances of a variety of operating systems may sharevirtualized hardware resources.

Network 430 includes one or more wired and/or wireless networks. Forexample, network 430 may include a cellular network (e.g., a fifthgeneration (5G) network, a long-term evolution (LTE) network, a thirdgeneration (3G) network, a code division multiple access (CDMA) network,etc.), a public land mobile network (PLMN), a local area network (LAN),a wide area network (WAN), a metropolitan area network (MAN), atelephone network (e.g., the Public Switched Telephone Network (PSTN)),a private network, an ad hoc network, an intranet, the Internet, a fiberoptic-based network, or the like, and/or a combination of these or othertypes of networks.

The number and arrangement of devices and networks shown in FIG. 4 areprovided as an example. In practice, there may be additional devicesand/or networks, fewer devices and/or networks, different devices and/ornetworks, or differently arranged devices and/or networks than thoseshown in FIG. 4. Furthermore, two or more devices shown in FIG. 4 may beimplemented within a single device, or a single device shown in FIG. 4may be implemented as multiple, distributed devices. Additionally, oralternatively, a set of devices (e.g., one or more devices) ofenvironment 200 may perform one or more functions described as beingperformed by another set of devices of environment 200.

FIG. 5 is a diagram of example components of a device 300. Device 300may correspond to user device 410 and/or platform 420. As shown in FIG.5, device 300 may include a bus 510, a processor 520, a memory 330, astorage component 540, an input component 550, an output component 560,and a communication interface 570.

Bus 510 includes a component that permits communication among thecomponents of device 300. Processor 520 is implemented in hardware,firmware, or a combination of hardware and software. Processor 520 is acentral processing unit (CPU), a graphics processing unit (GPU), anaccelerated processing unit (APU), a microprocessor, a microcontroller,a digital signal processor (DSP), a field-programmable gate array(FPGA), an application-specific integrated circuit (ASIC), or anothertype of processing component. In some implementations, processor 520includes one or more processors capable of being programmed to perform afunction. Memory 330 includes a random access memory (RAM), a read onlymemory (ROM), and/or another type of dynamic or static storage device(e.g., a flash memory, a magnetic memory, and/or an optical memory) thatstores information and/or instructions for use by processor 520.

Storage component 540 stores information and/or software related to theoperation and use of device 300. For example, storage component 540 mayinclude a hard disk (e.g., a magnetic disk, an optical disk, amagneto-optic disk, and/or a solid state disk), a compact disc (CD), adigital versatile disc (DVD), a floppy disk, a cartridge, a magnetictape, and/or another type of non-transitory computer-readable medium,along with a corresponding drive.

Input component 550 includes a component that permits device 300 toreceive information, such as via user input (e.g., a touch screendisplay, a keyboard, a keypad, a mouse, a button, a switch, and/or amicrophone). Additionally, or alternatively, input component 550 mayinclude a sensor for sensing information (e.g., a global positioningsystem (GPS) component, an accelerometer, a gyroscope, and/or anactuator). Output component 560 includes a component that providesoutput information from device 300 (e.g., a display, a speaker, and/orone or more light-emitting diodes (LEDs)).

Communication interface 570 includes a transceiver-like component (e.g.,a transceiver and/or a separate receiver and transmitter) that enablesdevice 300 to communicate with other devices, such as via a wiredconnection, a wireless connection, or a combination of wired andwireless connections. Communication interface 570 may permit device 300to receive information from another device and/or provide information toanother device. For example, communication interface 570 may include anEthernet interface, an optical interface, a coaxial interface, aninfrared interface, a radio frequency (RF) interface, a universal serialbus (USB) interface, a Wi-Fi interface, a cellular network interface, orthe like.

Device 300 may perform one or more processes described herein. Device300 may perform these processes in response to processor 520 executingsoftware instructions stored by a non-transitory computer-readablemedium, such as memory 330 and/or storage component 540. Acomputer-readable medium is defined herein as a non-transitory memorydevice. A memory device includes memory space within a single physicalstorage device or memory space spread across multiple physical storagedevices.

Software instructions may be read into memory 330 and/or storagecomponent 540 from another computer-readable medium or from anotherdevice via communication interface 570. When executed, softwareinstructions stored in memory 330 and/or storage component 540 may causeprocessor 520 to perform one or more processes described herein.Additionally, or alternatively, hardwired circuitry may be used in placeof or in combination with software instructions to perform one or moreprocesses described herein. Thus, implementations described herein arenot limited to any specific combination of hardware circuitry andsoftware.

The number and arrangement of components shown in FIG. 5 are provided asan example. In practice, device 300 may include additional components,fewer components, different components, or differently arrangedcomponents than those shown in FIG. 5. Additionally, or alternatively, aset of components (e.g., one or more components) of device 300 mayperform one or more functions described as being performed by anotherset of components of device 300.

The foregoing disclosure provides illustration and description, but isnot intended to be exhaustive or to limit the implementations to theprecise form disclosed. Modifications and variations are possible inlight of the above disclosure or may be acquired from practice of theimplementations.

As used herein, the term component is intended to be broadly construedas hardware, firmware, or a combination of hardware and software.

It will be apparent that systems and/or methods, described herein, maybe implemented in different forms of hardware, firmware, or acombination of hardware and software. The actual specialized controlhardware or software code used to implement these systems and/or methodsis not limiting of the implementations. Thus, the operation and behaviorof the systems and/or methods were described herein without reference tospecific software code—it being understood that software and hardwaremay be designed to implement the systems and/or methods based on thedescription herein.

Even though particular combinations of features are recited in theclaims and/or disclosed in the specification, these combinations are notintended to limit the disclosure of possible implementations. In fact,many of these features may be combined in ways not specifically recitedin the claims and/or disclosed in the specification. Although eachdependent claim listed below may directly depend on only one claim, thedisclosure of possible implementations includes each dependent claim incombination with every other claim in the claim set.

No element, act, or instruction used herein should be construed ascritical or essential unless explicitly described as such. Also, as usedherein, the articles “a” and “an” are intended to include one or moreitems, and may be used interchangeably with “one or more.” Furthermore,as used herein, the term “set” is intended to include one or more items(e.g., related items, unrelated items, a combination of related andunrelated items, etc.), and may be used interchangeably with “one ormore.” Where only one item is intended, the term “one” or similarlanguage is used. Also, as used herein, the terms “has,” “have,”“having,” or the like are intended to be open-ended terms. Further, thephrase “based on” is intended to mean “based, at least in part, on”unless explicitly stated otherwise.

What is claimed is:
 1. A method of displaying an augmented reality (AR)image, the method comprising: controlling an AR display to display theAR image as being at least partially overlaid over a view through amicroscope while the view is visible to a user of the microscope,wherein the AR image includes a cursor; receiving a user input from auser interface; setting an anchor point based on the user input;updating the AR image to include the anchor point; controlling the ARdisplay to display the updated AR image; detecting a motion of a slideof the microscope; adjusting the AR image by moving at least one fromamong the cursor and the anchor point within the AR image based on thedetected motion; and controlling the AR display to display the adjustedAR image.
 2. The method of claim 1, wherein the adjusted AR imagecomprises a line displayed between the anchor point and the cursor. 3.The method of claim 2, further comprising defining an area within the ARimage based on the line.
 4. The method of claim 3, wherein the anchorpoint is included in a plurality of anchor points, wherein the line isincluded in a plurality of lines determined based on the plurality ofanchor points, and wherein the area is bounded by the plurality oflines.
 5. The method of claim 3, wherein the area comprises a rectangle,and wherein the line is a diagonal of the rectangle.
 6. The method ofclaim 3, wherein the area comprises a circle, wherein the cursor islocated at a center of the circle, wherein the anchor point is locatedon a circumference of the circle, and wherein the line is a radius ofthe circle.
 7. The method of claim 3, wherein the adjusted AR imagecomprises a measurement determined based on the line, wherein themeasurement comprises at least one from among a length of the line, asize of the area, a color distribution of colors included in the area, acolor histogram of the colors included in the area, a frequencydistribution of the colors included in the area, a number of cellsincluded in the area, and a percentage of the area that includes apredetermined type of tissue.
 8. The method of claim 1, wherein the userinput comprises at least one from among a button press, a keyboardinput, a mouse input, and a voice input.
 9. The method of claim 1,further comprising capturing a plurality of image frames of the viewthrough the microscope, wherein a distance of the motion and a directionof the motion are determined based on a difference between a first pointin a first image frame of a plurality of image frames and a second pointin a second image frame of the plurality of image frames, the secondpoint corresponding to the first point, and wherein the adjusted ARimage is adjusted based on the determined distance and the determineddirection.
 10. The method of claim 1, further comprising storing animage comprising the view through the microscope at least partiallyoverlaid with the adjusted AR image.
 11. A device for displaying anaugmented reality (AR) image, the device comprising: an AR displayconfigured to display an AR image as being at least partially overlaidover a view through a microscope while the view is visible to a user ofthe microscope; at least one motion sensor configured to detect a motionof a slide of the microscope; a user interface configured to receive auser input; at least one memory configured to store program code; and atleast one processor configured to read the program code and operate asinstructed by the program code, the program code including: firstdisplaying code configured to cause the at least one processor tocontrol the AR display to display the AR image including a cursor;receiving code configured to cause the at least one processor to receivethe user input from the user interface; setting code configured to causethe at least one processor to set an anchor point based on the userinput; updating code configured to cause the at least one processor toupdate the AR image to include the anchor point; second displaying codeconfigured to cause the at least one processor to control the AR displayto display the updated AR image; detection code configured to cause theat least one processor to detect the motion of the slide from the motionsensor; adjusting code configured to cause the at least one processor toadjust the AR image by moving at least one from among the cursor and theanchor point within the AR image based on the detected motion; and thirddisplaying code configured to cause the at least one processor tocontrol the AR display to display the adjusted AR image.
 12. The deviceof claim 11, wherein the adjusted AR image comprises a line displayedbetween the anchor point and the cursor.
 13. The device of claim 12,wherein the program code further comprises defining code configured tocause the at least one processor to define an area within the AR imagebased on the line.
 14. The device of claim 13, wherein the anchor pointis included in a plurality of anchor points, wherein the line isincluded in a plurality of lines determined based on the plurality ofanchor points, and wherein the area is bounded by the plurality oflines.
 15. The device of claim 13, wherein the area comprises arectangle, and wherein the line is a diagonal of the rectangle.
 16. Thedevice of claim 13, wherein the area comprises a circle, wherein thecursor is located at a center of the circle, wherein the anchor point islocated on a circumference of the circle, and wherein the line is aradius of the circle.
 17. The device of claim 13, wherein the adjustedAR image comprises a measurement determined based on the line, whereinthe measurement comprises at least one from among a length of the line,a size of the area, a color distribution of colors included in the area,a color histogram of the colors included in the area, a frequencydistribution of the colors included in the area, a number of cellsincluded in the area, and a percentage of the area that includes apredetermined type of tissue.
 18. The device of claim 11, wherein the atleast one motion sensor comprises at least one image sensor configuredto capture a plurality of image frames of the view through themicroscope, wherein a distance of the motion and a direction of themotion are determined based on a difference between a first point in afirst image frame of the plurality of image frames and a second point ina second image frame of the plurality of image frames, the second pointcorresponding to the first point, and wherein the adjusted AR image isadjusted based on the determined distance and the determined direction.19. The device of claim 11, wherein the program code further comprisesstoring code configured to cause the at least one processor to store animage comprising the view through the microscope at least partiallyoverlaid with the adjusted AR image
 20. A non-transitorycomputer-readable medium storing instructions, the instructionscomprising: one or more instructions that, when executed by one or moreprocessors of a device for displaying an augmented reality (AR) image,cause the one or more processors to: control an AR display to displaythe AR image as being at least partially overlaid over a view through amicroscope while the view is visible to a user of the microscope,wherein the AR image includes a cursor; receive a user input from a userinterface; set an anchor point based on the user input; update the ARimage to include the anchor point; control the AR display to display theupdated AR image; detect a motion of a slide of the microscope; adjustthe AR image by moving at least one from among the cursor and the anchorpoint within the AR image based on the detected motion; and control theAR display to display the adjusted AR image.